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Educators are looking for ways to improve 
reading comprehension across content 
areas in late elementary years, especially 
among more disadvantaged students who 
increasingly fall behind. But first-year re- 
sults from this study of four grade 5 supple- 
mental reading comprehension curricula 
show no effects for three of the curricula 
and a negative effect for the fourth. 



In the late elementary grades — when students begin read- 
ing to learn, rather than learning to read — the cognitive 
demands on students increase. Students from disadvan- 
taged backgrounds often lack not only general vocabulary 
but the academic vocabulary that enables them to com- 
prehend what they are reading and thus acquire content 
knowledge. In addition, they often do not know how to 
organize and obtain knowledge from text in content areas, 
such as science and social studies. Instructional approaches 
for improving comprehension are not as well developed 
as those for improving decoding and fluency. Although 
multiple reading comprehension techniques for narrative 
text have been well demonstrated in small studies, there is 
little evidence on teaching reading comprehension within 
content areas. 

Improving the ability of disadvantaged students to read and 
comprehend text is also fundamental to the federal educa- 
tion policy aimed at closing the achievement gap between 
low- and high-achieving students. Title I of the No Child 
Left Behind Act of 2001 calls on educators to close the gap 
using approaches shown to be elfective through scientifi- 
cally based research. But such rigorous research is relatively 
scarce, making it difficult for educators to determine how 



best to use Title I funds to improve student outcomes. 
Identifying curricula that improve reading comprehension 
is part of this challenge. The Institute of Education Sciences 
has undertaken a rigorous evaluation of curricula designed 
to improve reading comprehension to meet that research 
challenge. 

The program 

Four supplemental curricula were selected in a competitive 
process to participate in a rigorous test of their elfectiveness 
beginning in the 2006/07 school year: 

• Project CRISS by CRISS. Project CRISS focuses on five 
keys to learning — background knowledge, purpose 
setting, author’s craft (which involves using text struc- 
ture to improve comprehension), active learning, and 
metacognition. The program is designed for daily use 
during language arts, science, or social studies. 

• ReadAbout by Scholastic. Primarily through a com- 
puter program, students learn reading comprehen- 
sion skills, such as author’s purpose, main idea, cause 
and effect, compare and contrast, summarizing, and 
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inferences, which they then apply to a selection of sci- 
ence and social studies books. 

• Read for Real by Chapman University and Zaner-Bloser. 
Teachers use a six- volume set of books to teach reading 
strategies students can use before, during, and after 
reading (such as previewing, activating prior knowl- 
edge, setting a purpose, establishing the main idea, 
using graphic organizers, and finding text structures). 
Each unit includes vocabulary, fluency, and writing 
activities. 

• Reading for Knowledge by the Success for All Founda- 
tion. Reading for Knowledge makes extensive use of 
cooperative learning strategies and a process called 
SQRRRL (survey, question, read, restate, review, learn). 

The study 

The study used a rigorous experimental design to assess the 
effects of four supplemental reading comprehension cur- 
ricula on the reading comprehension of grade 5 students in 
schools randomly assigned to either an intervention group 
that used one of the four study curricula or to a control 
group that continued with its usual curriculum. Control 
group teachers could, however, use other supplemental 
reading strategies. Intervention group teachers received 
training to implement the curriculum to which their school 
was assigned and then developed their own strategies for 
incorporating the assigned reading comprehension cur- 
riculum into their daily schedules and their core reading 
instruction — as the curricula were designed to supplement, 
not replace, the core curriculum. 

The experimental design ensures a strong basis for answer- 
ing the study’s two primary research questions: 

• What is the impact of the reading comprehension cur- 
ricula as a whole on student reading comprehension 
in grade 5, and how do the impacts of the individual 
curricula compare? 

• How are student, teacher, and school characteristics 
related to curriculum impacts? 

The study sample consisted of 10 districts, 89 schools, 268 
teachers, and 6,350 grade 5 students. Districts worked with 



the study team to identify schools for study participation 
that served low-income students and did not use any of the 
four selected curricula. 

The study focus was on testing curricula designed to 
improve comprehension of expository text. Outcomes were 
defined as the ability to comprehend such text generally 
and in two specific content areas, science and social studies. 
Impact estimates focused on student reading comprehen- 
sion test scores. Data on student outcomes were collected 
from student tests at the end of the 2006/07 school year on 
the Group Reading Assessment and Diagnostic Evaluation 
(GRADE) and on tests of comprehension of social studies 
and science text, using assessments developed by Educa- 
tional Testing Service for the study. In addition, classrooms 
were observed during the school year, and data were col- 
lected from teacher surveys, school information forms, stu- 
dent and school records, and the intervention developers. 

The findings 

The study’s main findings include information on cur- 
riculum implementation and the effects of each of the four 
tested curricula. 

Curriculum implementation 

During the summer and early fall of 2006 more than 90 
percent of teachers of each intervention curriculum were 
trained to use their curriculum: Read for Real, 91 percent 
of teachers; Reading for Knowledge, 96 percent; and Project 
CRISS and ReadAbout, 100 percent. More than half the 
teachers of each intervention curriculum reported feel- 
ing very well prepared by the training to implement their 
assigned curriculum: Reading for Knowledge, 56 percent of 
teachers; Project CRISS, 69 percent; ReadAbout, 72 percent; 
and Read for Real, 80 percent. 

At the time of the classroom observations in the spring of 
2007 more than 80 percent of teachers of each intervention 
curriculum reported using their assigned curriculum: Read 
for Real, 81 percent of teachers; Reading for Knowledge, 

83 percent; ReadAbout, 87 percent; and Project CRISS, 91 
percent. Classroom observation data showed that, on aver- 
age, teachers implemented 55-78 percent of the behaviors 
deemed important by the curriculum developers: Project 
CRISS, teachers implemented 78 percent of such behaviors; 
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ReadAbout, 71 percent; Reading for Knowledge, 58 and 
65 percent of the behaviors for the curriculum’s two types 
of instructional days; and Read for Real, 55 and 71 percent. 

To describe teacher practices, the study team constructed 
scales summarizing practices in three areas. Two of three 
scales were not statistically significantly different between 
the intervention and control groups: the reading strategy 
guidance and classroom management scale and the student 
engagement scale. Scores on the traditional interaction scale 
were statistically significantly lower for the intervention 
group than for the control group (an effect size of -0.52). 

Achievement effects 

Student achievement findings are based on the first year of 
data collected for the study (2006/07) for the first cohort of 
grade 5 students. Two types of impacts were analyzed. First, 
impacts were analyzed for each curriculum intervention; 
these impacts provide information on the effectiveness of 
each intervention. Second, the outcomes for students in 
all four intervention groups were combined and compared 
with the outcomes for all students in the control group; 
these findings address whether the use of these types of 
interventions, in general, improves comprehension. 

For the basic question of intervention effectiveness, findings 
showed that reading comprehension test scores were not 
statistically significantly higher in intervention schools 
than in control schools for three of the four tested curricula 
using a composite test score (an average of the scores on 
the three tests; see figure 1). The fourth tested curriculum, 
Reading for Knowledge, resulted in a statistically signifi- 
cant negative effect on student achievement for intervention 
schools, with an effect size of -0.14. 

For questions about for whom and under what conditions 
the interventions may be effective, findings revealed that 
reading comprehension test scores were statistically signifi- 
cantly lower in intervention schools than in control schools 
for some student subgroups, but no clear pattern emerged. 
For the combined intervention group as a whole, negative 
and statistically significant impacts were observed for the 
following subgroups on the identified tests: 

• Students with above-average baseline fluency, on the 
social studies reading comprehension test. 



FIGURE 1 

Effects of reading comprehension curricula on 
composite test scores for first cohort, 2006/07 
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*Statistically different from the control group at the .05 level. 

a. The combined treatment group shows the outcomes for students in all 
four intervention groups combined. These impacts are compared to those 
for students in the control group, providing information on the effective- 
ness of reading comprehension interventions more broadly rather than 
the specific impacts of any one intervention. 



• Students with baseline comprehension levels in the 
bottom third of the sample, on the GRADE and the 
composite measure of the three test scores. 

• Students in schools with an above-average concentra- 
tion of students eligible for the federal lunch program, 
on the composite measure of the three test scores. 

• Students in schools with a below-average concentration 
of English language learner students, on the composite 
measure of the three test scores. 

• Students whose teachers had more than five years of 
experience, on the composite measures of the three test 
scores. 

Several other subgroups were tested using teacher char- 
acteristics that could have been affected by the training 
teachers received, and therefore the findings should not be 
used to suggest a causal relationship. For the combined in- 
tervention group a negative, statistically significant impact 
on the composite test measure was found for students in 
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schools with a below-average school professional culture 
scale score. No significant impacts were found for sub- 
groups based on teachers’ past professional development or 
teaching efficacy. 

The next steps 

In the 2007/08 school year the study team collected a sec- 
ond year of data. To examine the extent to which interven- 
tion impacts are sustained over time, students from the 
study’s first year were followed for one more year using the 
same follow-up outcome measures. With a new cohort of 
grade 5 students, the study was repeated for three of the 
four interventions to assess whether the interventions are 
more effective after schools and teachers have had a year of 



experience using them. Results from year 2 of the study will 
be presented in a later report. 

For the full report, please visit: 

http://ies.ed.gOv/ncee/pubs/20094032/ 
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NCEE developed the Evaluation Briefs to offer short 
synopses of complex technical evaluation reports. This 
brief was not prepared by the study authors. 
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